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Developing an Effective Assessment Process to 
Support Institutional Effectiveness Efforts 

Abstract 

This paper describes a process and an organizational structure in which Institutional 
Effectiveness (IE) can thrive. When such a system is implemented, continuous quality 
improvement can be sustained for the long-term. Key concepts and philosophies found in the 
literature regarding continuous quality improvement and assessment in higher education are 
briefly discussed. Major factors, including competition in the academy, escalation of legislative 
accountability mandates and the heightened focus on customer satisfaction, that have been 
influential in the increased prioritization and the polarization of institutional effectiveness are 
also introduced. 

The primary focus of this paper is the general structure implemented at a major 
metropolitan research university that supports ongoing assessment activities. The overall 
organizational structure used for implementing this process is described. Additionally, a 
succinct explanation of the roles and responsibilities of key participants within the assessment 
organizational structure is presented for consideration. The paper also addresses ways to 
increase the practicality of sustaining accurate, complete, and thorough documentation of 
assessment efforts. 
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1.0 Introduction 

The processes of establishing, measuring and reviewing academic programs’ learning 
outcomes and administrative units’ operational objectives are an integral part of the institution- 
wide institutional effectiveness process that supports the Southern Association of Colleges and 
Schools (SACS) criteria for regional accreditation, as well as other accrediting bodies. While an 
intrinsic value of education lies in the love for teaching and learning, effectiveness in the 
preparation of students for successful careers is progressively becoming more important as a 
key measure of success within the higher education arena. The overwhelming expectation is 
that academia will prepare students to work, not only within academe, but will equip them with 
sufficient knowledge and skills that will enable them to perform in the business, technology, and 
service industries. 

Ewell (1999) stated that authentic, creative and action-oriented assessment approaches 
are ignited by two key factors: 

a) taken from the proper value perspective, assessment constitutes a powerful 
tool for collective improvement that is highly consistent with core academic 
values and b) infusion of the logic of assessment directly into classroom and 
curricular settings is perhaps the most powerful means we have at our disposal 
to transform the logic of pedagogy itself — from one-way instruction to 
collaboration and partnership, (pg. 147) 

Since the mid 1980s, many individuals in higher education questioned the need for a 
revitalized focus on quality in higher education and the applicability of quality concepts in the 
higher education system. The need to renew and strengthen the focus on quality stems from 
several sources. Lewis and Smith (1994) found that the reasons that were causing a change in 
attitude regarding the value of assessment included (1) reports and commentaries expressing 
distrust and dissatisfaction with the higher education system, (2) shifting trends of the student 
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body composition to older, married and working individuals, (3) increased competition stemming 
from the demands of students for value added education, (4) technological innovations and the 
reality of virtual campuses and (5) the reality of limited funding coupled with an expectation of 
accomplishing more. 

Despite the fact that those individuals who are involved in teaching and students’ 
learning believe that the quality of their educational product is evident without the requirement of 
conducting assessment, the most effective means for publicizing how well higher education is 
doing is to publish data that reflects what is actually being accomplished. Not only is this an 
effective response, it is becoming increasingly prevalent as a result of recent legislative laws 
that enforce accountability measures in order to receive public funding. Lewis and Smith (1994) 
contended that although academicians may not agree with the criticism, we achieve little by 
becoming defensive, closed minded and attacking. The more appropriate reaction is to 
embrace the philosophy of continuous quality, thereby enhancing the rich heritage of higher 
education. 

Banta (1993) explained that the overarching concern in higher education relative to 
measuring and continuous quality improvement is whether lasting improvements can be made 
in higher education through the adaptation of techniques that seem to be working in industry. 
The dialogue centers on the viability of using principles of continuous quality improvement from 
industry to effect positive results in the higher education culture when the culture is so distinctly 
different. 

According to Hubbard (1993) there are three quality concepts in terms of relevance for 
higher education that increases the chances for successful implementation of continuous quality 
improvement principles. The principles of parsimony (sharply focused goals, clear definitions of 
quality, fewer administrative layers, the distinction between the critical and the trivial), 
benchmarking (comparing against best practices) and customer satisfaction are essential to the 
quality movement. 
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Cornesky, McCool, Burns & Weber (1992) explained that as educators and higher 
education administrators, if we assume that everyone wants to do a good job, and that we work 
within processes and systems, then it naturally follows that our focus should be on continuously 
improving those processes and systems. If this is the commitment, then we can expect not only 
better quality results, but increased productivity as well. 

Having adequate support for assessment and a process for quality assurance of the 
assessment process are two critical areas of concern. This paper describes how the University 
of Central Florida (UCF) has established a system to support their assessment activities. 

Section 2 introduces the components of an assessment process and section 3 discusses the 
essential participants in the assessment process and needed support functions. Documentation 
of the assessment plans and results is often a necessary time-consuming activity and section 4 
describes the method currently employed by UCF. Section 5 contains concluding remarks. 

2.0 Components of an Assessment Process 

Assessment involves measuring outcomes and using those measurements to take 
actions to make improvements. The need to conduct assessment can come from a number of 
different sources, both internal and external to the university. Regional accreditation, program 
accreditation and program reviews all constitute some of the external drivers. Strategic 
planning and the desire to improve are internal drivers. 

After a determination has been made that there is a need to implement an assessment 
initiative within an institution, the next step is to determine the scope of the effort. The scope 
involves considering who the key players will be, determining the assessment cycle (annual, 
every three years, etc.), determining the method of submission of assessment plans and results, 
devising the review process, documenting use of results, developing a system for archiving 
assessment activity, developing a system for recognizing successful assessment efforts and 




6 



Developing an Effective Assessment Process 6 



maintaining the continual development and dynamic implementation of assessment policies and 
guidelines. 

The University of Central Florida is a metropolitan research university that was 
established in 1963 and is located just outside Orlando. UCF has grown from 2,600 students to 
36,000 students in 38 years and is currently classified as “Research Intensive.” UCF has the 
second largest undergraduate enrollment in the State and employs approximately 1000 faculty 
and 3500 staff members. It has five colleges (Arts and Sciences, Business Administration, 
Education, Engineering and Computer Science, and Health and Public Affairs), and an Honor’s 
College. 

The assessment process was established at UCF in 1996 in response to “criticisms" 
from SACS. During the 1996-97 academic year, UCF adopted the Nichols model (Nichols, 

1991 & 1995) as the institution-wide method for conducting and documenting the assessment of 
all academic programs and administrative units. At that time, each of the approximately 350 
academic programs and administrative units developed mission statements, three to five 
objectives or learning outcomes, and at least two measurement approaches per objective or 
learning outcome. Over the following two years, each program established appropriate 
methods (surveys, standardized tests, etc.) for assessing the objectives or outcomes and began 
using the assessment results to make changes to academic programs and administrative 
operations. 

Today, the mission, objectives and measures for all programs and units are published on 
the web at http://132.170.220.225/oeas/ for public access. The assessment plans and results 
for all of the programs undergo a thorough annual review by the Divisional Review Committee 
(DRC) in the colleges and divisions, followed by a final review by the University Assessment 
Committee (UAC). The UAC specifically checks to ensure that the mission and objectives are 
consistent with the mission of the institution, that the objectives support the mission, and that 
the measures will provide useful results to improve the programs. The Divisional Review 
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Committee and the UAC also review the results from the assessment process to ensure that the 
use of results to identify needed improvements is properly documented. 

3.0 Essential Participants and Functions Supporting the Assessment Process 

A key to the successful implementation of an assessment process is having an 
appropriate support organization. The critical players in this assessment process at UCF are 
the 

• University Assessment Committee (UAC), Office of Operational Excellence & 
Assessment Support (OEAS), Divisional Review Committees (DRC), and 

• Assessment coordinators for academic programs and directors of administrative 
units. 

The integration of the roles of each of these major participants is vital to the attainment of 
institutional effectiveness and assessment goals. Following is a description of each primary 
function and the general composition of the organization. 

3.1 The Role and Function of the University Assessment Committee (UAC) 

The President of UCF established the University Assessment Committee in 1997 to 
provide quality assurance for UCF’s institutional effectiveness process. The role of the 
University Assessment Committee is to foster and promote a university culture that values 
assessment and the benefits of continuous quality improvement. The UAC’s function extends to 
enhancing the assessment process through the development of policy and guidelines for the 
institutional effectiveness initiative. The organization encourages assessment training and 
education, develops criteria for assessment plans and results, determines the submission cycle 
and schedule, plans and designs the review process, oversees the Divisional Review 
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Committees’ work, provides technical expertise university-wide and reports to the President on 
an annual basis the state of institutional effectiveness and assessment. 

The UAC ‘s charge to implement a meaningful review process entails ensuring that all 
final reviews of academic programs and administrative units are conducted by the committee, 
and that feedback regarding the strengths and weaknesses of plans and documentation of 
results are communicated in a timely manner to the appropriate individuals. The UAC also 
assists Academic Affairs in earmarking available funds for academic programs and 
administrative units wishing to implement effectiveness measures that the UAC believes to have 
merit, such as standardized testing. The committee also recommends names of faculty and 
staff to Academic Affairs to attend assessment conferences and institute, which is also 
underwritten by allocated funds. 

The President appoints the UAC members to the university committee. The Chair of the 
UAC is a faculty member and is compensated through a reduction in other duties (e.g., a 
“course release”) and a summer stipend. Other members serve on a voluntary basis as part of 
their service contribution to the university. 

The UAC membership consists of one member in each of the following areas: 

• Chair (faculty member) 

• College of Arts & Sciences 

• College of Business Administration 

• College of Education 

• College of Engineering & Computer Science 

• College of Health & Public Affairs 

• Administration & Finance 

• Information Technologies and Resources 

• Sponsored Research 
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• Student Development & Enrollment Services 

• Academic Affairs 

• University Relations 

• President’s Divisionln general, the members from the colleges are Assistant or 
Associate Deans and the members from the administrative areas are leaders of assessment 
within their college or division. 

3.2 The Role and Function of Operational Excellence & Assessment Support (OEAS) 

The office of Operational Excellence and Assessment Support was established in March 
2000 as part of UCF’s reorganization of its Institutional Research function. The role of OEAS is 
to provide support to academic programs and administrative units in a broad range of activities. 
These activities include preparing for regional and program accreditation; coordinating 
assessment support; coordinating the development and administration of institutional level 
surveys; coordinating survey design and analysis for colleges, academic programs, 
administrative divisions and units; conducting process analyses; engaging in environmental 
scanning and conducting special studies. 

Additionally, OEAS’s role involves providing administrative support to the UAC by 

1. maintaining the Institutional Effectiveness (IE) website, 

2. coordinating assessment meetings, 

3. preparing UAC meeting minutes, 

4. maintaining assessment archives, 

5. coordinating UAC level communication to the Divisional Review Committees, 
faculty and staff and 

6. assisting the UAC in preparing annual institutional level assessment reports. 
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The assessment support staff members in the OEAS office conduct assessment clinics 
(office hours) and provide consultations with faculty and staff who need assistance with 
developing assessment plans for their programs and units. The office also provides workshops 
and training for larger groups. The survey support staff assists academic programs and 
administrative units in the design, implementation, and analysis of surveys to support their 
assessment efforts. 

The OEAS staff includes a director, assistant director, survey and statistical studies 
coordinator, senior statistician, process analysis and special studies coordinator, administrative 
assistant, computer support specialist, secretary (part-time) and five graduate assistants. The 
office operates on an annual budget of approximately $500,000. 

3.3 The Role and Function of the Divisional Review Committee (DRC) 

Divisional Review Committees (DRCs) were established in Spring of 2001. The DRC 
members are appointed by the UAC with the full knowledge and concurrence of the dean of the 
college or the vice president of the administrative division. The purpose of each committee is to 
further advance assessment expectations of the university to faculty and academic 
administrators on a college and divisional level. This is typically accomplished by the DRC 
members on a more personal basis and with a less formal approach. 

Members of the committee have the advantage of sharing the same culture of that of 
their colleagues, and consequently are in a more superior strategic position to impact the 
assessment process. Within their respective areas, DRC members have received training from 
OEAS and are adequately positioned to assist programs and units with the successful 
completion and submission of assessment plans, assist with the necessary revisions that may 
be required and assist with the appropriate documentation and revision of assessment results. 

Additionally, the DRC members support the university and the UAC by conducting 
internal reviews of assessment plans and results within their college or division. At a minimum, 
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the DRCs utilize the UAC approved criteria in conducting all reviews. However, if additional 
criteria are developed at the college or divisional level, advance notice of the criteria is provided 
and sanctioned by the UAC. 

For quality control purposes, the DRC is required to invite members of the Planning & 
Evaluation (an umbrella unit within the Academic Affairs division that includes OEAS) staff to 
attend meetings as observers or as active participants. At least one Planning & Evaluation staff 
member must participate in DRC meetings during which an actual review of assessment plans 
and results occur. 

The nine divisional committees are made up of faculty members who are usually the 
assessment coordinators for one or more academic programs within the college and of 
administrators and staff who are responsible for developing assessment plans within the 
administrative divisions. These individuals have over time demonstrated their involvement in 
the assessment process and that they have a genuine interest in the institutional effectiveness 
and assessment effort. A UAC member or an appointed committee member chairs each 
divisional committee. 

3.4 The Role and Function of Academic Program Assessment Coordinators 

According to the SACS criteria, 

“educational activities of an institution include teaching, research and 
public service. Planning and evaluation for these activities must be systematic, 
broad based, interrelated and appropriate to the institution. The institution must 
define its expected educational results and describe its methods for analyzing the 
results. 

The institution must (1) establish a clearly defined purpose appropriate to 
collegiate education, (2) formulate educational goals consistent with the 
institution’s purpose, (3) develop and implement procedures to evaluate the 
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extent to which these educational goals are being achieved and (4) use the 
results of these evaluations to improve educational programs, services and 
operations. 

The evaluation of academic programs should involve gathering and 
analyzing both quantitative and qualitative data that demonstrate student 
achievement. Measures to evaluate academic programs and general education 
may include the following: evaluation of instructional delivery; adequacy of 
facilities and equipment; standardized tests; analysis of theses, portfolios, and 
recitals; completion rates; results of admissions tests for students applying to 
graduate or professional schools; job placement rates; results of licensing 
examinations; evaluations by employers; follow-up studies of alumni; and 
performance of student transfers at receiving institutions. The institution must 
evaluate its success with respect to student achievement in relation to purpose, 
including, as appropriate, consideration of course completion, state licensing 
examinations, and job placement rates.” 

The academic colleges assign faculty to participate in the assessment initiative at the 
program level. The faculty that are assigned to the role of assessment coordinator for a specific 
academic program are required to collaborate with other faculty within that discipline to develop 
an assessment plan for submission to the UAC. The appointed assessment coordinator may be 
responsible for this task each year, or it may be a rotating assignment. The individual assigned 
to this task is expected to work with the DRC in order to fulfill university assessment 
requirements. 

3.5 The Role and Function of Administrative Unit Directors 

According to the SACS criteria, 
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“in addition to providing evidence of planning and evaluation in its 
educational program, the institution must demonstrate planning and evaluation in 
its administrative and educational support services. For each administrative and 
educational support service unit, the institution must (1) establish a clearly 
defined purpose which support the institution’s purpose and goals, (2) formulate 
goals which support the purpose of each unit, (3) Develop and implement 
procedures to evaluate the extent to which these goals are being achieved in 
each unit and (4) use the results of the evaluations to improve administrative and 
education support services. 

Each unit, in its planning and evaluation processes, should consider 
internal and external factors and develop assessment methods that will yield 
information useful to the planning processes of that unit.” 

In general, the directors of the various units within the administrative divisions are 
responsible for coordinating and developing the assessment plans for a specific unit. It is the 
responsibility of that individual to work with other staff within the unit to develop a workable 
assessment plan for submission to the UAC. The individual assigned to this task is expected to 
work with the DRC in order to fulfill university assessment requirements. 

3.6 A Team Approach to Assessment Support 

The University Assessment Committee, the office of Operational Excellence and 
Assessment Support, and the Divisional Review committee work collaboratively to results in an 
efficient submission and review process. These three entities provide three distinct services. 
They (1) assist academic programs and administrative units with the development of plans and 
results, (2) conduct thorough reviews of plans and results to provide important feedback to units 
and (3) assist units in the submission process using the Institutional Effectiveness (IE) Website. 
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The assessment cycle is concurrent in that at any one given time there are three years 
under consideration. For example, within the 2001-2002 year, during summer through spring 
semesters, assessment is being conducted. In the spring semester 2002, assessment plans 
for 2002-2003 are submitted. This timeframe is considered Phase I of the cycle. Phase II 
occurs during the fall of the year (October - December). During this phase academic programs 
and administrative units report on assessment results from the previous assessment year 
(2000-2001). In the spring semester (February through March), academic programs and 
administrative units submit their assessment plans for the upcoming year. OEAS facilitates 
Assessment Clinics to provide necessary training and education in the development of plans. 
Then the Divisional Review Committees (DRC) and the UAC conduct their reviews. The DRC 
provides interim feedback to unit for necessary revisions and the UAC makes the final approval 
decisions based on the UAC representative’s consensus recommendations. The following 
statuses are used: Approved, minor revisions/no resubmission required, minor 
revisions/resubmission required, Major Modifications/resubmission required. 

OEAS communicates the UAC’s final review status and pertinent feedback to academic 
programs and administrative units. Additionally, units via the IE Website can access review 
feedback. A formal summary letter is sent by OEAS to vice presidents and deans, announcing 
official UAC approval status of assessment plans. All assessment plans must be finalized and 
approved by June. Approved plans are made available on the IE Website. 

During the fall semester, units submit assessment results from past year (e.g. In October 
2002 units submit results from 2001-2002 year). The DRC, UAC and OEAS coordinate during 
the review process to provide feedback to units for necessary revisions. All reviews are 
completed by January of the following year. Unlike the assessment plans, assessment results 
are not made available to the general public through the IE website. The assessment process 
is specifically designed to encourage programs and units to measure and track their 
performance in order to find areas for improvement. It was determined that if the results were 
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made public, programs and units would be less inclined to measures outcomes where they may 
not look as good. 

4.0 Documenting the Assessment Plans, Results and Reviews 

The IE Website was developed in January 2002. Prior to the development of the 
website, UCF used a paper submission process for three years (1997-1999). At that time each 
program submitted a five-column table with the following five elements: 

1. Mission statement 

2. Objectives and outcomes 

3. Measures 

4. Results 

5. Use of Results 

These tables were submitted once a year and then reviewed by the UAC. The review process 
often came too late for the programs to make changes in their next plan. In 1999, UCF split the 
submission process into two phases: Phase 1: Submission (in the Spring semester) of the 
assessment plan for the upcoming year and Phase 2: Submission (in the Fall semester) of the 
documentation of the assessment results for the past year. At that time, WORD templates 
(which became the basis for the IE website) were also designed for the submission and review 
processes. Programs and units submitted their assessment plans and results as email 
attachments. The following pages include the Word templates that were used for electronic 
submission of assessment plans and results. The review forms that were used by OEAS, the 
Divisional Review Committee and the University Assessment Committee are also included in 
the following pages. 
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Recently implemented IE website includes links for developing and submitting 
assessment plans, developing and submitting assessment results, reviewing the assessment 
plans and documentation of results of assessment plans, statistical reports on submission rates, 
and statuses on the official UAC review of assessment plans and results. In an effort to improve 
user friendliness and provide assessment support for specific colleges and administrative 
divisions, changes on the website are made on an ongoing basis. The IE Website has many 
advantages when compared to the hard copy and electronic submission. Some of the 
advantages for users include (1) ready access to past and current plans, (2) editing existing 
plans versus creating new plans, and (3) the ability to submit and resubmit with ease. For the 
UAC advantages include (1) an accurate database of academic programs and administrative 
units with contact information who are required to conduct assessment, (2) the capability to 
monitor the submission rate at any time, (3) a more efficient review process, (4) the ability to 
share approved submissions with the University community and (5) a more efficient archival 
system. Academic programs and administrative units are encouraged to use the IE Website at 
http://1 32. 1 70.220.225/oeas/ for purposes of gathering information and directions on how to 
develop and submit viable assessment plans and appropriate, useable documentation of 
assessment results. However, for individuals who choose not to use the website to submit their 
plans, there is an option of submitting electronic documents via email. No programs or units 
chose this option. 

4.1 Positive Results of Use of the Website for Assessment 

The assessment database, which supports the web pages for the assessment 
submission process, yields useful data that will provide insight into the direction and trends of 
assessment for the University. Other capabilities include making the development of 
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assessment plans and reporting of results of assessment “best practices” available to those who 
have the responsibility for assessment. These “best practices” will provide examples of 
excellent learning outcomes (academic programs), objectives (administrative units), 
measurement approaches, data results, plans for use of the results and documentation of 
implementation of changes for improvement purposes. Additionally, the UAC will be better 
equipped to explore a reward system for programs and units that are committed to making 
improvements based on assessment by using available data. Overall, the level of openness 
and receptivity to the new website has been very positive. 

4.2 Assessment and Website Challenges and Ideas 

Although many positive consequences exist as a result of the use of the IE Website, 
challenges associated with the assessment process and the newly implemented website are still 
a reality. Some of these challenges include (1) faculty and administrators frustration with a new 
method of submission (i.e., the “learning curve”), (2) some faculty’s dilemma with assessing 
learning outcomes because they view teaching as an art, (3) the perception that there is no 
benefit associated with the assessment process, (4) the assessment process is too much work, 
(5) a perceived lack of support from academic leaders, (6) a fear that the administration will use 
the results from the assessment process to evaluate programs or units, and (7) some 
differences in assessment philosophies between faculty and administrators. 

There are a number of things that can be done to enhance the assessment culture. For 
example, the development of a reasonable and viable reward system and the provision of 
opportunities for faculty and administrators may make working on assessment a more attractive 
proposition. Integrating the assessment processes to support regional accreditation, program 
accreditation, strategic planning, and performance reviews would help to streamline the entire 
process. Coordination of data needs through one source and the use of the data and 
information collected for all types of assessment for multiple purposes are also strategies for 
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improving the assessment culture. The efficient administrations of surveys by integrating 
individual surveys into more comprehensive institutional level surveys that are conducted and 
analyzed at the institutional level help reduce the workload on the programs and units. The 
publication of assessment success stories could also lead to a more effective system. 

5.0 Conclusion 

The institutional effectiveness and assessment initiatives support the university’s 
strategic direction of operational excellence and quality. The process that is currently in place at 
UCF is one that enhances both institutional effectiveness and assessment. Not only does the 
process enhance these two important initiatives, it is vital to the overall success of university as 
it strives to improve its programs and services. 

The university’s culture is progressively moving in the direction of appreciating the need 
to conduct assessment annually and realizing the importance of gathering meaningful data for 
improvement purposes. All of the steps within the assessment process (i.e., submission of 
assessment plans and results, and the review of assessment plans and results), the key 
organizations and participants (i.e., University Assessment Committee, Divisional Review 
Committees, Operation Excellence and Assessment Support office, and the Assessment 
Coordinators) and the functions of each of the groups (i.e., assist, train, and review) are 
essential to a smooth transition to a quality culture that operates at its highest level. 
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